Questions and instruments
Problematic Questions: lack of context
Even people who are familiar with the deal may not know the formal name, the “Iran nuclear deal” might be more common.
If feasible, a description of the policy may be more useful than just the name.
Problematic Questions: double-barreled items
- This is an example of a double-barreled question. I might want to spend more on welfare and less on national defense, or vice versa.
- At a minimum, we probably want to split these up. (The General Social Survey asks a whole series of spending questions like this)
Problematic Questions: non-exclusive categories
- These response options aren’t clearly mutually exclusive: I may have participated in the past but be unwilling to do so in the future. Or vice-versa.
- Splitting this up or adding a full range of options could be better.
Problematic Questions: ambiguous time frames
- Respondents can interpret the relevant time frame differently. Lifetime? In the last week?
- Specifying some time frame can give you more consistent answers.
Problematic Questions: context and recall
- Respondents may not see themselves as victims, may forget minor crimes, or may not immediately think of even major crimes as “criminal”.
- Surveys like the NCVS use examples and ask about specific aspects of each crime.
Problematic Questions: loaded language
- Explicitly referencing loaded terms can make people less willing to give honest answers, especially on sensitive questions.
- Respondents may genuinely not consider themselves drug abusers.
Problematic Questions: leading
- Mentioning one party here is likely to nudge respondents toward blaming Democrats, and the response options would fail to capture the possibility of assigning equal blame to both parties.
Social Desirability: voting
As many as half of non-voters will claim to have voted in survey questions.
(Hanmer, Banks, and White 2014) randomly gave subtle and unsubtle hints that researchers might check voter roles.
Social Desirability: voting
- Perhaps unsurprisingly, the desire to avoid being caught in a lie overwhelmed social desirability.
Social Desirability: list-experiments
Respondents are randomly assigned to receive only non-sensitive items, or non-sensitive items plus one sensitive question.
They only say how many items they agree with.
The difference in the average count of items is used to estimate support.
Fence siting
Some respondents may always pick the “neutral” option, if one is available.
One option is to eliminate the neutral option all together, but there’s a trade-off here: some people genuinely don’t have an opinion!
Acquiescence/naysaying: F-scale
- The F-scale was developed out of an early effort to understand the psychology of rank and file fascists.
- 30 agree/disagree items, with agreement indicating more authoritarian views.
- The lack of balance raised serious questions about bias: acquiescence bias would probably inflate the estimates of authoritarian attitudes.
Careless responding
- In a 2022 online opt-in survey, Pew found that 5% of adults said they were licensed to operate a nuclear submarine.
- The actual number would round to 0%
Racial resentment
Basic problem: professions of “old-fashioned” racism declined precipitously after the Civil Rights movement.
But most Americans continued to oppose programs designed to improve racial equality.
Symbolic racism (or racial resentment, or modern racism) is a proposed explanation for this phenomenon.
Social Desirability
Everyone wants to be liked. Respondents will give answers that make them look good. Maybe even subconsciously.